<html>
<head><meta charset="utf-8"><title>playground issues? · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html">playground issues?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="231593446"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231593446" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Gilcher <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231593446">(Mar 24 2021 at 07:32)</a>:</h4>
<p>Does the playground have issues lately? I regularly get my runs killed.</p>



<a name="231623843"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231623843" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231623843">(Mar 24 2021 at 12:22)</a>:</h4>
<p>Everything looks reasonably OK in terms of cpu and memory; I don't think we currently track success and lack thereof though, so that may have changed. <span class="user-mention" data-user-id="116155">@Jake Goulding</span> likely can give a better assessment there than I, but I'm not particularly aware of any problems.</p>



<a name="231627780"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231627780" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231627780">(Mar 24 2021 at 12:54)</a>:</h4>
<p>I do think it's something that's been happening more frequently — <a href="https://github.com/integer32llc/rust-playground/issues/668">https://github.com/integer32llc/rust-playground/issues/668</a></p>



<a name="231627813"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231627813" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231627813">(Mar 24 2021 at 12:54)</a>:</h4>
<p>Based on my own anecdata as well.</p>



<a name="231627884"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231627884" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231627884">(Mar 24 2021 at 12:55)</a>:</h4>
<p>Like <strong>simulacrum</strong>  mentioned, whenever I've poked at the CPU / memory stats, nothing seems wildly off.</p>



<a name="231628060"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231628060" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231628060">(Mar 24 2021 at 12:56)</a>:</h4>
<p>We do not track success/failure (although that's in my long term goals list; but I don't even know what data collector to use). The best we have is an access log.</p>



<a name="231628199"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231628199" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231628199">(Mar 24 2021 at 12:57)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> do we have a dashboard or anything that tracks machine-level stats over time? If so, we could see if the playground server has been getting more and more traffic (read: CPU usage) or anything.</p>



<a name="231628386"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231628386" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231628386">(Mar 24 2021 at 12:58)</a>:</h4>
<p>We do for CPU and memory, though history is currently just two weeks. It doesn't look like there's been any spike though and we're generally at an average of around 40-50% CPU, though with short peaks of up to 100%</p>



<a name="231628513"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231628513" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231628513">(Mar 24 2021 at 12:59)</a>:</h4>
<p>IIRC, the docker container we launch imposes memory limits, maybe we should relax those a bit? The memory statistics I can see suggest we have plenty of headroom right now</p>



<a name="231628561"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231628561" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231628561">(Mar 24 2021 at 12:59)</a>:</h4>
<p>I believe SIGKILL is how memory oom usually shows up in docker</p>



<a name="231631312"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231631312" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231631312">(Mar 24 2021 at 13:16)</a>:</h4>
<p>If it were memory spikes I would expect it to be consistently killed though :/ this is only happening sometimes</p>



<a name="231638744"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231638744" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231638744">(Mar 24 2021 at 14:03)</a>:</h4>
<p>Well, it depends on how the oom killing in docker works. If it's not an edge-trigger but rather polling, I could easily see the detection <em>usually</em> going unnoticed, as it slips by in just under the polling timer firing</p>



<a name="231690480"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231690480" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Gilcher <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231690480">(Mar 24 2021 at 19:02)</a>:</h4>
<p><span class="user-mention silent" data-user-id="232545">Joshua Nelson</span> <a href="#narrow/stream/242791-t-infra/topic/playground.20issues.3F/near/231631312">said</a>:</p>
<blockquote>
<p>If it were memory spikes I would expect it to be consistently killed though :/ this is only happening sometimes</p>
</blockquote>
<p>Hm, I'm not sure. Given that I see this immediately, i think there's some credibility to that  theory.</p>



<a name="231690607"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231690607" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Gilcher <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231690607">(Mar 24 2021 at 19:03)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/242791-t-infra/topic/playground.20issues.3F/near/231628513">said</a>:</p>
<blockquote>
<p>IIRC, the docker container we launch imposes memory limits, maybe we should relax those a bit? The memory statistics I can see suggest we have plenty of headroom right now</p>
</blockquote>
<p>I see this on examples of all sizes, especially without any external crates or so, so i'm not sure it's a headroom problem.</p>



<a name="231691195"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231691195" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231691195">(Mar 24 2021 at 19:07)</a>:</h4>
<p>(I think we always build with all external crates due to how playground is setup). but yeah, hard to say</p>



<a name="231790703"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231790703" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231790703">(Mar 25 2021 at 12:23)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> do we track memory usage of rustc and or cargo over time? Maybe it’s a matter of just using more little by little and now it’s bumping into our caps.</p>



<a name="231790768"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231790768" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231790768">(Mar 25 2021 at 12:23)</a>:</h4>
<p>We track rustc, I don't think we've had any major changes lately though. <a href="https://perf.rust-lang.org/?start=&amp;end=&amp;absolute=true&amp;stat=max-rss">https://perf.rust-lang.org/?start=&amp;end=&amp;absolute=true&amp;stat=max-rss</a></p>



<a name="231790788"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231790788" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231790788">(Mar 25 2021 at 12:24)</a>:</h4>
<p>Maybe cargo has, though.</p>



<a name="231790844"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231790844" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231790844">(Mar 25 2021 at 12:24)</a>:</h4>
<p>There’s also things like <a href="https://github.com/integer32llc/rust-playground/issues/608">https://github.com/integer32llc/rust-playground/issues/608</a></p>



<a name="231790859"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231790859" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231790859">(Mar 25 2021 at 12:24)</a>:</h4>
<p>But I think we should raise the cap (how was it chosen?) and see if that helps things; we seem to have plenty of headroom right now so there's little risk to doing so I imagine?</p>



<a name="231790885"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231790885" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231790885">(Mar 25 2021 at 12:24)</a>:</h4>
<p>Where we already ran out of memory for bigger things.</p>



<a name="231791139"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231791139" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231791139">(Mar 25 2021 at 12:26)</a>:</h4>
<blockquote>
<p>how was it chosen?</p>
</blockquote>
<p>xkcd-random.png</p>



<a name="231791215"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231791215" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231791215">(Mar 25 2021 at 12:27)</a>:</h4>
<p>Probably focusing on my little t2.micro instance</p>



<a name="231791280"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231791280" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231791280">(Mar 25 2021 at 12:27)</a>:</h4>
<p><a href="https://github.com/integer32llc/rust-playground/blob/d03009c20002b21d76f2f01b1efdd2749a0559f1/ui/src/sandbox.rs#L450">https://github.com/integer32llc/rust-playground/blob/d03009c20002b21d76f2f01b1efdd2749a0559f1/ui/src/sandbox.rs#L450</a></p>



<a name="231791341"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231791341" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231791341">(Mar 25 2021 at 12:28)</a>:</h4>
<p>256 MB should be enough for anybody</p>



<a name="231792139"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/231792139" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#231792139">(Mar 25 2021 at 12:34)</a>:</h4>
<p>Maybe I’ll make it into an environment variable tonight.</p>



<a name="233126125"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/233126125" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#233126125">(Apr 05 2021 at 01:19)</a>:</h4>
<p>Memory-related seems likely:</p>
<div class="codehilite"><pre><span></span><code>~# dmesg | grep &#39;out of memory: Killed process&#39; | wc
     53     901    9937
</code></pre></div>



<a name="234338957"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234338957" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234338957">(Apr 13 2021 at 14:48)</a>:</h4>
<p>rustfmt is finally building, so the Playground CI is flowing again. I've doubled the memory limits (512 RAM, 640 swap), so let's watch and hope! <span aria-label="fingers crossed" class="emoji emoji-1f91e" role="img" title="fingers crossed">:fingers_crossed:</span></p>



<a name="234510985"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234510985" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234510985">(Apr 14 2021 at 14:46)</a>:</h4>
<p>I do note that the playground only has 2 cores and the load average is at <code>6.83, 6.77, 4.8</code>. Do we know how easily the number of cores could be changed?</p>



<a name="234511045"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234511045" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234511045">(Apr 14 2021 at 14:47)</a>:</h4>
<p>Maybe it's time to finally have a pool of playground servers</p>



<a name="234511573"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234511573" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234511573">(Apr 14 2021 at 14:49)</a>:</h4>
<p>I don't think we would need to jump to a pool; moving playground to a slightly larger server might make sense, but ultimately it's not clear to me that is necessary right now</p>



<a name="234511638"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234511638" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234511638">(Apr 14 2021 at 14:50)</a>:</h4>
<p>the load has looked pretty constant</p>



<a name="234511668"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234511668" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234511668">(Apr 14 2021 at 14:50)</a>:</h4>
<p><a href="/user_uploads/4715/sLapxDPGKJ8xne2Hs5Pc-u1U/image.png">image.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/sLapxDPGKJ8xne2Hs5Pc-u1U/image.png" title="image.png"><img src="/user_uploads/4715/sLapxDPGKJ8xne2Hs5Pc-u1U/image.png"></a></div>



<a name="234511689"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234511689" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234511689">(Apr 14 2021 at 14:50)</a>:</h4>
<p>that's the last 7 days cpu</p>



<a name="234511733"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234511733" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234511733">(Apr 14 2021 at 14:50)</a>:</h4>
<p>(which is actually really interesting why core 0 and core 1 are always ... the same)</p>



<a name="234524444"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234524444" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234524444">(Apr 14 2021 at 15:51)</a>:</h4>
<p>I feel like that... can't be right. How can the load average be &gt;2 (now: <code>3.30, 3.86, 4.51</code>) on a two core machine but the CPU looks pretty well below 50%</p>



<a name="234525003"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234525003" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234525003">(Apr 14 2021 at 15:54)</a>:</h4>
<p>But, top does say</p>
<div class="codehilite"><pre><span></span><code>%Cpu(s): 22.7 us, 11.9 sy,  0.0 ni, 64.6 id,  0.7 wa,  0.0 hi,  0.0 si,  0.2 st
</code></pre></div>



<a name="234531986"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234531986" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234531986">(Apr 14 2021 at 16:36)</a>:</h4>
<p>The "load" is not the amount of running tasks, but the amount of tasks that is running and the amount of tasks waiting to be scheduled combined. So for example if on a single core processor one task is running and 10 are waiting to be scheduled next, the load is 11.</p>



<a name="234549437"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234549437" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234549437">(Apr 14 2021 at 18:22)</a>:</h4>
<p>Right, but why would there be processes wanting to run but the CPU not at 100% (aka not running them)?</p>



<a name="234549855"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234549855" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234549855">(Apr 14 2021 at 18:24)</a>:</h4>
<p>Waiting on disk i/o for example?</p>



<a name="234549909"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234549909" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234549909">(Apr 14 2021 at 18:25)</a>:</h4>
<p><a href="/user_uploads/4715/uLqSqfCVYv7uIBKaTxBxO2uU/image.png">image.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/uLqSqfCVYv7uIBKaTxBxO2uU/image.png" title="image.png"><img src="/user_uploads/4715/uLqSqfCVYv7uIBKaTxBxO2uU/image.png"></a></div>



<a name="234550242"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234550242" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234550242">(Apr 14 2021 at 18:27)</a>:</h4>
<p>Hmm; if that's the case, then improving speed would be better suited to trying to reduce disk IO costs</p>



<a name="234550497"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234550497" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234550497">(Apr 14 2021 at 18:28)</a>:</h4>
<p>But I don't generally see anything as <code>D</code> when watching via top</p>



<a name="234550603"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234550603" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234550603">(Apr 14 2021 at 18:29)</a>:</h4>
<p>The playground is Ubuntu 18 and an older docker, so it's always possible that newer versions have improved perf as well</p>



<a name="234550935"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234550935" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234550935">(Apr 14 2021 at 18:31)</a>:</h4>
<p>Yeah, we should upgrade the machine regardless (just to a newer AMI)</p>



<a name="234554015"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234554015" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234554015">(Apr 14 2021 at 18:53)</a>:</h4>
<p>What’s that process look like? Is it “just” a config change and reboot?</p>



<a name="234554124"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234554124" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234554124">(Apr 14 2021 at 18:53)</a>:</h4>
<p>Someday we will get something automated that I can also use for my playground instance.</p>



<a name="234554130"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234554130" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234554130">(Apr 14 2021 at 18:53)</a>:</h4>
<p>Hm, it's probably easier to just provision an entirely new instance - I think it usually wants to swap out the disk used, for example</p>



<a name="234556249"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234556249" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234556249">(Apr 14 2021 at 19:06)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> do you know if this is feasible:</p>
<blockquote>
<p>something automated that I can also use for my playground instance</p>
</blockquote>
<p>Something I could put into simpleinfra (or in the playground repo itself...) but also target my own machines / EC2 credentials. That way I could try out the config for <code>play.integer32.com</code> and ensure that it's the same as the real playground.</p>



<a name="234556338"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234556338" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234556338">(Apr 14 2021 at 19:06)</a>:</h4>
<p>I'ven't done enough with terraform to gauge such a thing</p>



<a name="234556410"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234556410" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234556410">(Apr 14 2021 at 19:06)</a>:</h4>
<p>We have some ansible scripts which seem better suited to the configuration; terraform to my knowledge is not well suited to configuring instances post boot</p>



<a name="234556447"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234556447" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234556447">(Apr 14 2021 at 19:07)</a>:</h4>
<p>But you could do something like what we have for the monitoring instance</p>



<a name="234556485"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234556485" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234556485">(Apr 14 2021 at 19:07)</a>:</h4>
<p>(and bastion, I think)</p>



<a name="234556832"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234556832" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234556832">(Apr 14 2021 at 19:09)</a>:</h4>
<p>terraform == make the machine; ansible == set up the machine?</p>



<a name="234557038"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234557038" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234557038">(Apr 14 2021 at 19:10)</a>:</h4>
<p>I have a smidgen of ansible experience, but it was a few years ago now :-)</p>



<a name="234557082"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234557082" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234557082">(Apr 14 2021 at 19:11)</a>:</h4>
<p>yeah, that's my understanding</p>



<a name="234568711"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234568711" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234568711">(Apr 14 2021 at 20:24)</a>:</h4>
<blockquote>
<p>Waiting on disk i/o for example?</p>
</blockquote>
<p>Newer kernels have <code>/proc/pressure/io</code> for that. Recent versions of <code>htop</code> can also be configured to show that. Or just <code>watch cat /proc/pressure/*</code>. It's better than looking at load averages since it splits IO, CPU and memory subsystem contention.</p>



<a name="234574966"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234574966" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234574966">(Apr 14 2021 at 21:03)</a>:</h4>
<p>we should definitely use ansible to configure the playground instance</p>



<a name="234575159"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234575159" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234575159">(Apr 14 2021 at 21:04)</a>:</h4>
<p><span class="user-mention" data-user-id="116155">@Jake Goulding</span> I'm wondering, what is running on the playground machine and how is play deployed?</p>



<a name="234575297"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234575297" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234575297">(Apr 14 2021 at 21:04)</a>:</h4>
<p>if it's just a docker image behind nginx it'd be actually really simple to port it to ansible, as we already have roles ready for all of that</p>



<a name="234580117"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234580117" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234580117">(Apr 14 2021 at 21:32)</a>:</h4>
<p><span class="user-mention silent" data-user-id="330154">The 8472</span> <a href="#narrow/stream/242791-t-infra/topic/playground.20issues.3F/near/234568711">said</a>:</p>
<blockquote>
<p><code>watch cat /proc/pressure/*</code></p>
</blockquote>
<div class="codehilite"><pre><span></span><code>tail  /proc/pressure/*
==&gt; /proc/pressure/cpu &lt;==
some avg10=31.54 avg60=17.47 avg300=22.47 total=497882726599

==&gt; /proc/pressure/io &lt;==
some avg10=1.21 avg60=0.88 avg300=0.99 total=35939772899
full avg10=0.39 avg60=0.40 avg300=0.40 total=19590670619

==&gt; /proc/pressure/memory &lt;==
some avg10=0.00 avg60=0.00 avg300=0.00 total=477442351
full avg10=0.00 avg60=0.00 avg300=0.00 total=179537271
</code></pre></div>



<a name="234581053"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234581053" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234581053">(Apr 14 2021 at 21:39)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/playground.20issues.3F/near/234575297">said</a>:</p>
<blockquote>
<p>if it's just a docker image behind nginx it'd be actually really simple to port it to ansible, as we already have roles ready for all of that</p>
</blockquote>
<p>Oh you sweet summer child.</p>
<p>"Deploying" is a cronjob that runs every X (15?) minutes, running <code>docker pull</code> for the stable/beta/nightly/miri/rustfmt/clippy containers and syncing the assets and server binary from a S3 bucket -&gt; local. It then restarts the server (whether or not anything changed).</p>
<p>Original server setup <em>should</em> basically be "install docker, create a file-backed partition for the temporary files, create a daemon, configure nginx to forward to the server"</p>



<a name="234586123"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234586123" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234586123">(Apr 14 2021 at 22:23)</a>:</h4>
<p>a prometheus node_exporter has a enabled-by-default module for pressure information. And there's a <a href="https://grafana.com/grafana/dashboards/12949">dashboard</a> by yours truly</p>



<a name="234586568"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20issues%3F/near/234586568" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20issues.3F.html#234586568">(Apr 14 2021 at 22:27)</a>:</h4>
<p><span class="user-mention" data-user-id="116155">@Jake Goulding</span> could we have the playground itself be inside a docker container?</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>